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Abstract of EP1 092973 

A surface inspection system (10) is provided for 
detecting defects on a surface of a workpiece. 
The system (10) includes a diffused liglit source 
(22) for emitting an elongated line of light (23) 
onto the surface of the workpiece (25), a movable 
member (26) for translating the workpiece (25) in 
relation to the light source (22), an imaging 
device (24) positioned at a vantage point such 
that the line of light (23) is within its field of 
observation for capturing two or more sets of 
image data representative of a portion of the 
surface of the workpiece, and a data structure for 
storing model data, where the model data is 
indicative of the spatial relationship between the 
surface of the workpiece (25) and the observation 
plane of the imaging device (24). The surface 
inspection system further includes an anomaly 
detection module (28) adapted to receive the 
image data from the imaging device (24) and for 
identifying at least one potential surface defect in 
the image data, and a defect tracking module 
(30) connected to the data structure and the 
anomaly detection module (28) for tracking the 
potential surface defect from the first set of image 
data to the second set of image data by using 
said model data, thereby assessing if the 
potential surface defect constitutes a defect in the 
surface of the workpiece (25). 
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(54) A system and method for detecting defects in a surface of a workplece 



(57) A surface inspection system (1 0) is provided for 
detecting defects on a surface of a workpiece. The sys- 
tem (10) includes a diffused light source (22) for emit- 
ting an elongated line of light (23) onto the surface of the 
workpiece (25), a movable member (26) for translating 
the workpiece (25) in relation to the light source (22), an 
imaging device (24) positioned at a vantage point such 
that the line of light (23) is within its field of observation 
for capturing two or more sets of image data represent- 
ative of a portion of the surface of the workpiece, and a 
data structure for storing model data, where the model 
data Is Indicative of the spatial relationship between the 
surface of the workpiece (25) and the observation plane 



of the imaging device (24). The surface inspection sys- 
tem further includes an anomaly detection module (28) 
adapted to receive the image data from the imaging 
device (24) and for identifying at least one potential sur- 
face defect in the image data, and a defect tracking 
module (30) connected to the data structure and the 
anomaly detection module (28) for tracking the potential 
surface defect from the first set of image data to the sec- 
ond set of image data by using said model data, thereby 
assessing if the potential surface defect constitutes a 
defect in the surface of the workpiece (25). 
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inspection process. 



Bacicqround and Summary of the invention 

[0001 ] The present invention relates generally to an s 
automated surface inspection system and, more partic- 
ularly, a system and method for detecting defects on a 
painted surface of a workplece. 
[0002] Demand for higher quality has pressed man- 
ufacturers of mass produced articles, such as automo- io 
tive vehicles, to employ automated manufacturing 
techniques that were unheard of when assembly line 
manufacturing was first conceived. Today, robotic equip- 
ment is used to assemble, weld, finish, gauge and test 
manufactured articles with a much higher degree of is 
quality and precision than has been heretofore possible. 
Computer-aided manufacturing techniques allow 
designers to graphically conceptualize and design a 
new product on a computer workstation and the auto- 
mated manufacturing process ensures that the design 20 
Is faithfully carried out precisely according to specifica- 
tion. 

[0003] Quality control is also an Important compo- 
nent of the automated manufacturing process. For 
instance, rather than employing human inspectors, 25 
automated surface inspection systems are used to per- 
form repetitive visual inspection of a workpiece In order 
to detect flaws in the surface of a workpiece. 
[0004] An imaging device is typically used in an 
automated surface inspection system to capture frames 30 
of image data representative of a portion of the surface 
of the workpiece. To determine what constitutes a sur- 
face defect, the surface Inspection system compares 
the location of a potential surface defect from frame to 
frame. In the conventional case, the surface inspection 35 
system assumes that the inspection surface is substan- 
tially planar, and thus the potential surface defect is 
moving at the same or proportional rate (In relation to 
the Imaging device) at which the workpiece Is being 
translated by the movable member. However, for con- 40 
toured inspection surfaces, the potential surface defect 
does not travel at the same rate in relation to the imag- 
ing device as the velocity of the movable member. As a 
result, conventional surface Inspection systems are 
unable to accurately compare the location of a potential 45 
defect between frames, and thus may be unable to iden- 
tify surface defects in contoured inspection surfaces. 
[0005] Therefore, It Is desirable to provide an auto- 
mated system and method for detecting defects In a 
contoured surface of a workpiece. To the extent that the so 
inspection surface is painted, it is also desirable that the 
surface inspection system distinguish surface flaws 
caused by dirt, pinholes or scratches from the surface 
roughness of the paint (i.e., "orange peel") on the work- 
piece. It would further be advantageous If such a sys- ss 
tern were able to characterize the defects by size so that 
only those body parts having defects larger than a cer- 
tain threshold size would need to be rejected by the 



Summary of tha invention 

[0006] In accordance with the present invention, a 
surface Inspection system Is provided for detecting 
defects on a surface of a workpiece. TTie surface 
inspection system Includes a diffused light source for 
emitting an elongated line of light onto the surface of the 
workpiece, a movable member for translating the work- 
piece in relation to the light source, an Imaging device 
positioned at a vantage point such that the line of light is 
within Its field of observation for capturing two or more 
sets of image data representative of a portion of the sur- 
face of the workpiece, and a data structure for storing 
model data which is indicative of the spatial relationship 
between the surface of the workpiece and the observa- 
tion plane of the imaging device. The surface Inspection 
system further includes an anomaly detection module 
for identifying at least one potential surface defect in the 
image data and a defect tracking module connected to 
the data structure and the anomaly detection module for 
tracking the potential surface defect from the first set of 
image data to the second set of Image data, thereby 
assessing if the potential surface defect constitutes a 
defect in the surface of the workpiece. 
[0007] For a more complete understanding of the 
invention, its objects and advantages, reference may be 
had to the following specification and to the accompany- 
ing drawings. 

Brief Description of the Drawings 
[0008] 

Figure 1 is a perspective view of a surface inspec- 
tion station on an automotive assembly line in 
accordance with the present invention; 

Figure 2 is a block diagram of the basic compo- 
nents of the surface inspection system of the 
present invention; 

Figures 3A-3B illustrates image processing opera- 
tions which are applied by the anomaly detection 
subsystem of the present invention; 

Figures 4A and 48 are block diagrams illustrating 
the operations performed by the defect tracking 
subsystem in accordance with the present inven- 
tion; 

Figure 5 Is a diagram showing how the velocity of a 
potential surface defect In relation to the imaging 
device varies from the velocity of the conveyor sys- 
tem; 

Figure 6 illustrates an exemplary 30 facet model for 
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an automotive vehicle body in accordance with the 
present Invention; 

Figure 7 is a diagram showing an exemplary 
numeric assignment for each pixel location near the 
predicted pixel tor a potential surface defect which 
is then used to determine a tracl<ing strength for 
that potential surface defect in accordance with the 
present invention; and 

Figure 8 Is a diagram illustrating how Identified sur- 
face defects are mapped back to a model of the 
vehicle body In accordance with the present inven- 
tion. 

Description of the Preferr ed Embodiment 

[0009] An exemplary surface Inspection system of 
the type commonly employed in vehicle assembly lines 
Is shown in Figure 1. The surface inspection system 10 
Includes a conveyer system 1 2 for carrying vehicle bod- 
ies 14 through various assembly stations on the assem- 
bly line. A mounting frame 1 6 surrounds the conveyor 
system 12 and provides a mounting position for at least 
two light sources 22. In addition, the mounting frame 16 
provides a plurality of other mounting positions for a 
series of imaging devices 24. Communication cables, 
which are not specifically shown in Fig. 1, couple the 
light sources 22 and the imaging devices 24 to one or 
more computing devices 18. Although the Invention is 
not limited to automotive applications, an exemplary use 
for the surface inspection system of the present Inven- 
tion would be in an automotive assembly plant. 
[0010] Figure 2 illustrates the basic components 
associated with the surface inspection system 10 in 
accordance with the present invention. The surface 
inspection system 10 generally includes two diffused 
light sources 22, one or more Imaging devices 24, a 
movable member 26, an anomaly detection subsystem 
28, and a defect tracking subsystem 30. 
[0011] Two light sources 22 are used to emit two 
elongated lines of light 23 onto the surface of a work- 
piece 25. Preferably, each light source 22 Is a fluores- 
cent lamp diffused through a thin sheet of diffusing film. 
An exemplary light source is manufactured by Plastic 
Films Company It is also envisioned that a baffle (not 
shown) may be positioned between each of the light 
sources 22. Although two light sources are presently 
preferred, this Is not intended as a limitation of the 
invention. On the contrary, one or more light sources 
may be suitably used. 

[0012] At least one imaging device 24 is used to 
capture image data representative of a portion of the 
surface of the workpiece. The imaging device 24 is posi- 
tionable at a vantage point such that the line of light 23 
is within the device's field of observation. It should be 
noted that the length of the light source 22 is such that 
the field of observation of the imaging device 24 is fully 



illuminated by the light source 22. An exemplary imag- 
ing device 24 may be one of a variety of CCD cameras 
manufactured by Pulnix Corporation. While the follow- 
ing description is provided with reference to one imag- 

5 Ing device, it is readily understood that a plurality of 
imaging devices may be used to inspect the same or dif- 
ferent portions of the surface of the workpiece. 
[0013] Generally, the intensity of light source along 
with the imaging device and its lens setting are chosen 

10 so that surface defects such as dirt, craters, solvent 
pops, and scratches create contrast in the captured 
image which is greater than the surface roughness of 
the paint on the workpiece (commonly referred to as 
'orange peel"). The light Intensity gradient from the 

15 elongated lines of light Is such that it provides a means 
for emphasizing the defect contrast relative to the back- 
ground. Thus, potential surface defects are detected by 
the light reflected back to and captured by the imaging 
device 24. Since the surface roughness of the paint is 

20 highly non-repeatable and varies from surface to sur- 
face on a vehicle body (e.g., it being generally higher on 
a vertical surface than on a horizontal surface), the 
anomaly detection subsystem 28 provides an adaptive 
technique that de-emphasizes the image distortions 

25 caused l3y the surface roughness of the paint. 

[001 4] If the surface of the workpiece is painted with 
a high reflectivity color (e.g., white paint), then the inten- 
sity of the light source 22 may saturate the surface of 
the workpiece such that the imaging device 24 is unable 

30 detect surface defects from the reflected light. One 
skilled in the art will readily recognize that the surface 
inspection system 1 0 is operative to dim or brighten the 
light sources 22 based on the color of the inspected sur- 
face. The intensity of the light source is determined 

35 through empirical testing for each color of the inspected 
surface which may be inspected by the surface inspec- 
tion surface 10. 

[0015] In operation, the workpiece 25 is coupled to 
a movable member 26 which translates the workpiece 

40 25 in relation to the light sources 22. More specifically, 
the movable member 26 travels in a direction substan- 
tially perpendicular to the lines of light from the light 
sources 22. As a result, the inspection surface of the 
workpiece passes through the lines of light such that the 

45 imaging device 24 is able to capture Image data repre- 
sentative of a portion of the surface of the workpiece. In 
addition, the movable member 26 may travel at varying 
rates of speed. As will be apparent to one skilled in the 
art, the rate of speed may be adjusted so that the imag- 

50 ing device 24 is able to capture more or less image data 
for a particular portion of the workpiece as it moves 
through the lines of light from the light sources 22. 
Although a conveyor system is presently preferred for 
use as the movable member 26, this Is not intended as 

55 a limitation of the broader aspects of the present inven- 
tion. On the contrary, other types of movable members 
may be used In conjunction with the present invention. 
[0016] An anomaly detection subsystem 28 and a 
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defect tracking subsystem 30 reside on one or more of 
the computing devices (not shown) associated with the 
surface inspection system 10. As will be more fully 
explained below, the anomaly detection subsystem 28 
is adapted to receive image data from the imaging 
device 24 and operative to identify potential surface 
defects in the Image data; whereas the defect tracking 
subsystem 30 is operative to track the potential surface 
defects, thereby assessing if any one potential surface 
defect constitutes a defect in the surface of the work- 
piece. One skilled in the art will readily recognize that 
the anomaly detection subsystem 28 and the defect 
tracking subsystem 30 may be implemented using com- 
mercially available hardware and software components 
such as those available from Matrox Electronics Sys- 
tems Ltd. 

[0017] A more detailed description of a preferred 
embodiment of the surface inspection system 10 is pro- 
vided in relation to Figures 3-6. In order to identify 
potential surface defects, numerous sets of image data 
are received and processed by the anomaly detection 
subsystem 28. For instance, a first set of image data 32 
and a second set of image data 34 are received by the 
anomaly detection subsystem 28 in Figure 3A. Various 
Image processing techniques are then applied to each 
pair of image frames. 

[0018] First, an absolute subtraction 36 occurs 
between the first image frame 32 and the second image 
frame 34. This operation removes background noise 
from the imaged surface that are stationary relative to 
the imaging device. The resulting image frame 38 is 
indicative of the differences between the two sets of 
Image data. Absolute subtraction is preferred because it 
enhances all of the anomalies encapsulated in both sets 
of Image data. 

[0019] Next, the resulting image frame 38 under- 
goes a thresholding function 40. This operation reduces 

noise resulting from the subtraction operation. Moreo- 
ver, the thresholding function 40 converts the integer 
representation of the image data into a binary represen- 
tation of the image data. In other words, rather than rep- 
resenting the pixel data as an integer between 0 and 
255, the pixel data is stored as either a 0 or 1 . This con- 
version to a binary image frame reduces the data by 
eight fold, and thus increases the speed of the remain- 
ing image processing operations. The resulting binary 
image frame is shown at 42. 

[0020] "Orange peel" is randomly varying structure 
noise that exists on a painted surface. "Orange peel" is 
part of any painting process on a metal surface and is 
caused by the viscosity of the paint. Rather than soften- 
ing the sharp edges of the lines of light from the light 
sources to minimize the effects of "orange peel" in the 
image data, the anomaly detection subsystem 28 fur- 
ther includes a filtering process for removing the 
"orange peel" anomalies found in the image data. As will 
be apparent to one skilled in the art. this filtering proc- 
ess is particularly applicable to painted inspection sur- 



faces, whereas the remainder of the present invention 
applies to any bare metal inspection surfaces. 
[0021] The "orange peel" filtering process begins 
with a copy of the first set of image data 32 and the sec- 

5 ond set of image data 34 as shown in Figure 3B. First, 
an edge detection operation 52 as is well known in the 
art is used to identify the edges of the lines of light in 
each set of image data. The resulting images are shown 
at 54 and 56. As previously explained, a thresholding 

10 function 58 is applied to each of these images 54 and 56 
to reduce noise as weil reduce the amount of image 
data. As will be apparent to one skilled in the art, the 
selection of thresholding is such that the resultant 
image from step 58 contains predominantly light edges, 

15 but not defects that are farther from the edges of the 
lines of light. 

[0022] Next, the binary image data frames are 
dilated using a dilation operation 60 as is well known in 
the art. In this case, the dilation operation enhances the 

20 thickness of the detected edges of the lines of lights. 
The dilated images are shown at 62 and 64. Each of the 
above operations are performed on both sets of image 
data as shown in Figure 3B. Each set of image data 62 
and 64 are combined using an OR operation. The "OR" 

25 operation 66 ensures that the "orange peel" anomalies 
from each set of image data are represented in the 
resulting composite image data. The resulting compos- 
ite image data is shown at 68. 
[0023] Referring to Figure 30, this composite image 

30 data 68 is inverted (using a "NOT" operation) 70 to gen- 
erate an orange peel mask 72. The orange peel mask 
72 predominantly represents the "orange peel" on the 
painted surface of the workpiece and can be applied as 
a filter to the binary image frame 42. To do so, the 

35 orange peel mask 72 is combined with the binary image 
frame 42 using an "AND" operation 74. In this way, the 
"orange peel" is removed from the image data. The 
resultant image frame 76 identifies only potential sur- 
face defects in the workpiece and can be used as input 

40 to the defect tracking subsystem 30. 

[0024] Generally, the defect tracking subsystem 30 
receives potential surface defect information from the 
anomaly detection subsystem 18 and tracks the posi- 
tion of each potential surface defect as it moves within 

45 the reference frame of the imaging device. If the position 
of the potential surface defect is accurately predicted, 
then the potential defect may be classified as a defect in 
the surface of the workpiece. On the other hand, if the 
position of the potential surface defect is not accurately 

50 predicted, then it may be assumed that the potential 
defect constitutes random noise and may be disre- 
garded. 

[0025] In order to track a potential surface defect 
from frame to frame, the defect tracking subsystem 30 
55 requires at least two resulting image frames from the 
anomaly detection subsystem 18. As will be more fully 
described below, a first resulting image frame will yield 
potential surtece defects which were captured in the 
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second set of Image data and a second resulting Image 
frame will yield potential surface defects which were 
captured in a third set of image data. As will be apparent 
to one skilled In the art, the second set of image data 
and the third set of Image data serve as input to the 5 
anomaly detection subsystem 18, thereby resulting In 
the second resultant image frame received from the 
anomaly detection subsystem 18. A more detailed 
description of the defect tracking subsystem 30 Is pro- 
vided in relation to Figures 4A and 4B. 10 
[0026] First, a dilation operation 94 as Is well known 
In the art is applied to a first resultant image frame 
received from the anomaly detection subsystem 28. In 
this case, the dilation operation merges anomalies that 
are within close proximity of each other (e.g., within a is 
couple of pixels). As a result, the image processing 
associated with a subsequent connectivity operation is 
reduced. 

[0027] The image frame is then overlaid with a grid 
96 which divides the larger anomalies into smaller 20 
anomalies. For instance, a 30x30 pixel grid may be used 
to divide up the larger anomalies. As will be more fully 
explained below, this operation helps to accurately pre- 
dict the state of a vector that represents the anomaly 
(which would otherwise be very sensitive to defect 25 
shape changes) and to accurately identify certain struc- 
tural features (e.g., a door handle on a side body panel) 
on the vehicle body 

[0028] The amount of potential surface defect data 
is further reduced through the use of a connectivity or 30 
"blobbing" operation 98. Rather than processing a col- 
lection of Individual pixel data, the connectivity opera- 
tion forms shaped objects from the related anomaly 
pixels. To do so, this operation connects pixels that are 
similar and within one pixel distance from each other. 35 
The connectivity operation Is performed within the 
30x30 pixel grid. If a "blob" extends outside the grid, 
then it becomes multiple blobs. 
[0029] Since the image frame received from the 
anomaly detection subsystem 28 encapsulates image 40 
data from two different image frames, each potential 
surface defect appears twice in the image frame 
received from the anomaly detection subsystem 18. A 
pair elimination operation 1 00 is therefore used to elim- 
inate half of each paired blobs. To do so, a polarity 45 
check is applied to each set of blobs to eliminate at least 
one of the blobs. For Instance, a bounding box is drawn 
around each of the blobs. The gray level deviation within 
the bounding box is computed for the most recent image 
frame and the previous image frame. If the deviation so 
from the latest image frame is greater than the deviation 
in the previous image frame (and some predefined 
value), then the blob Is determined to be from the latest 
image frame. Lastly, the blob from the latest image 
frame is retained whereas the blob from the previous 55 
image frame is discarded. Accordingly, each remaining 
blob corresponds to potential surface defects which 
were captured in the second set of image data 34 
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received by the anomaly detection subsystem 18. 
[0030] Next, a centrold is calculated 102 for each 
remaining blob. The centroid will be used by the tracking 
operation 106 to predict the position of the potential sur- 
face defect from frame to frame. In addition, a compact- 
ness factor (i.e., area/perimeter^) is computed for each 
blob. Thus, the compactness factor describes the shape 
of the blob. Each blob may be characterized by a state 
vector, where the state vector is defined as a list of nor- 
malized attributes that describe the location and shape 
of an anomaly It is envisioned that other morphological 
features (e.g., perimeter, area, a region of interest, etc.) 
may also be computed and used as attributes to defined 
the state vector for each blob. The normalization is nec- 
essary because the information associated with loca- 
tion would have a different range compared with the 
information associated with the shape. These normal- 
ized set of values for location and shape attributes 
describe each blob. A state vector for each blob in the 
Image frame is then passed along to the tracking oper- 
ation 106. 

[0031] Referring to Figure 4B. the tracking opera- 
tion 106 receives a state vector for each blob and tracks 
the state of each vector as it moves within the reference 
frame of the imaging device. The match between the 
predicted state vector and its "true" state is quantified by 
the probability of both being the same. Since the geo- 
metric configuration for each blob is relatively invariant 
between frames, the compactness factor is used to 
identify a blob from one frame to the next frame. The 
centroid for each blob is then used track the location of 
a blob from frame to frame, thereby determining a pre- 
dicted location probability for the blob. 
[0032] In order to predict the position of the poten- 
tial surface defect, the tracking operation 106 must first 
compute the velocity at which the defect is travelling In 
relation to the imaging device. For instance, if the 
inspection surface of the workpiece is substantially pla- 
nar, then the potential surface defect Is travelling at the 
same rate (in relation to the imaging device) as the 
workpiece is being moved by the conveyor system. 
Since the workpiece is being moved by the conveyor 
system at a known constant velocity, the position of or 
distance traveled by the potential surface defect can be 
predicted from frame to frame. As will be apparent to 
one skilled in the art, the time occurring between frames 
is known because the imaging device is capturing 
image data at a known frequency 
[0033] However, for an inspection surface which is 
not substantially planar, the rate at which the potential 
surface is travelling in relation to the imaging device 
does not correspond to the velocity of the conveyor sys- 
tem. Due to the contour of the inspection surface 1 20, a 
potential surface defect 122 will appear to be moving 
faster (or slower) than the rate of the conveyor system 
124 as shown in Figure 5. Therefore, the defect tracking 
subsystem 30 utilizes model data indicative of the spa- 
tial relationship between the inspection surface of the 
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workpiece and the observation plane of the imaging 
device in order to determine the velocity of the potential 
surface defect in relation to the imaging device. 
[0034] A CAD model of an automotive vehicle body 
provides the spatial relationship between the inspection 
surface of the workpiece and the observation plane of 
the imaging device. A typical CAD model of a vehicle 
body Is constructed using 5,000 to 10,000 facets. For 
purposes of the present invention, the vehicle body 130 
is modeled using approximately 30 facets as shown in 
Figure 6. Although the invention is not limited to a sim- 
plified model of the vehicle body, it Is readily understood 
that a 30 facet model provides an adequate approxima- 
tion of the vehicle body as well as translates into a man- 
ageable amount of model data. Model data for each 
facet is then used to determine the spatial relationship 
between the observation plane of the imaging device 
and the inspection surface of the workpiece. 
[0035] As the workpiece passes through the obser- 
vation plane of the imaging device, the tracking opera- 
tion 106 accesses the model data for the facet which 
corresponds to the portion of the surface being 
inspected by the Inspection system 10. In this way. the 
tracking operation 106 is able to transform the velocity 
of the workpiece Into a velocity at which the potential 
surface defect is travelling in relation to the imaging 
device. The tracking operation 106 then uses this veloc- 
ity to predict the pixel position of the potential surface 
defect from frame to frame. Depending on the accuracy 
of this prediction, the tracking operation 106 assesses 
whether a potential surface defect constitutes a defect 
In the surface of the workpiece. 
[0036] Of course, this velocity transformation proc- 
ess must be done when the inspection surface corre- 
sponds to a new facet on the vehicle body. It is further 
envisioned that the surface inspection system 10 is 
operative to access different model data for different 
types of vehicle bodies. In addition, the frequency of the 
imaging device may be synchronized with the velocity of 
the conveyor system, thereby further simplifying the 
velocity transformation process. 
[0037] To assess what constitutes a defect, the 
tracking operation 106 computes a tracking strength for 
each potential surface defect as it moves between 
Image frames. The tracking strength is based on the 
actual location of the potential surface defect relative to 
the predicted location for that potential surface defect 
and the similarity in compactness between the state 
vectors representing the potential surface defects. Fig- 
ure 7 Illustrates how a numeric value is assigned to 
each pixel In close proximity of the pixel 1 12 which cor- 
responds to the predicted location of the potential sur- 
face defect. If the actual location of the potential surface 
defect 114 falls within one of these pixels, then that 
numeric value is used as a basis for the tracking 
strength. Because any particular potential surface 
defect may occur in three or more image frames, the 
tracking operation further maintains a cumulative track- 



ing strength value for each potential surface defect. This 
cumulative tracking strength value can then be com- 
pared to some predetermined threshold value. It Is envi- 
sioned that the threshold value is based on the number 

5 of times the position for any one potential surface defect 
was predicted within the viewing area of the imaging 
device. Finally, each potential surface defect can be 
classified as a surface defect or not based on whether 
its cumulative tracking strength exceeds the threshold 

10 value. A surface defect may also be characterized 
based on its cumulative tracking strength and/or the 
number of observations within the viewing area of the 
imaging device. 

[0038] Returning to Figure 4B, the defect tracking 

15 subsystem 30 includes at least two additional post- 
processing operations. First, a mapping operation 108 
as is well known in the art can be used to map any iden- 
tified surface defects back onto a model of the vehicle 
body. A camera model provides a numerical formulation 

20 that enables accurate description in the physical world 
of a pixel on the imaging device. In this case, the cam- 
era model incorporates the physical principles that dic- 
tate the formation of an image in the imaging device. 
Using this camera model, a ray is drawn through a 

25 potential surface defect such that it passes through a 
surface on the vehicle body as shown in Figure 8. 
[0039] Model data for each facet is translated along 
with the vehicle body. This translation is dictated by the 
measurement of position of the vehicle body. Intersec- 

30 tlon of this ray with a facet provides the location of the 
defect on the vehicle body. A vector summation of this 
ray with the translation vector associated with a facet on 
the vehicle body provides a method for accurately pre- 
dicting the position of the defect after car translation. 

35 Again, the camera model is used to calculate the posi- 
tion of the defect in the image after translation. The 
accuracy of the prediction depends on the accuracy of 
the facet model data and thus the translation vector. It 
should be noted that a velocity calculated using the 

40 above procedure is not limited to a single direction of 
travel for a surface defect along a non-planar surface of 
the vehicle body. Identified surface defects can also be 
displayed on a CRT or other type of display device 
which is associated with the computing device 18 of the 

45 surface inspection system 10. 

[0040] A masking operation 1 10 is then used to fil- 
ter out any known structural features of the vehicle body. 
In this case, since the structural features (e.g., a door 
handle on a side body panel) are identified as surface 

so defects by the surface inspection system, the masking 
operation 110 ensures that these features are not misl- 
dentifled as surface defects. 

[0041] While the Invention has been described in its 
presently preferred form, it will be understood that the 
55 invention is capable of modification without departing 
from the spirit of the invention as set forth in the 
appended claims. 
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Claims 

1. A surface inspection system for detecting defects 
on a surface of a workpiece (25, 120), in particular 
for detecting defects on a surface of a vehicle bo6y 
(14) within a motor vehicle manufacturing system, 
characterized by: 

at least one diffused fight source (22) for emit- 
ting an elongated line of light (23) onto the sur- 
face of the workpiece (25, 120); 
a movable member (26), in particular a con- 
veyer system (1 2), for translating the workpiece 
(25, 120) in relation to said light source (22); 
an imaging device (24) for capturing two or 
more sets of image data (32, 34) representa- 
tive of a portion of the surface of the workpiece 
(25. 120), said Imaging device (24) having a 
field of observation and being positionable at a 
vantage point such that the line of light (23) is 
within the field of observation; 
a data structure for storing model data indica- 
tive of a spatial relationship between said por- 
tion of the surface of the workpiece (25, 120) 
and the observation plane of said imaging 
device (24) and; 

an anomaly detection subsystem (28) adapted 
to receive at least a first set of Image data (32) 
and a second set of image data (34) from said 
imaging device (24), said anomaly detection 
subsystem (28) identifying at least one poten- 
tial surface defect (1 14, 122) in said first set of 
image data (32) and said second set of image 
data (34); and 

a tracking subsystem (30) connected to said 
data structure and said anomaly detection sub- 
system (28) for tracking the potential surface 
defect (114, 122) from said first set of Image 
data (32) to said second set of image data (34) 
using said model data, thereby assessing if the 
potential surface defect (114, 122) constitutes 
a defect in the surface of the workpiece (25, 
120). 

2. The surface inspection system of Claim 1 , charac- 
terized in that said tracking subsystem (30) opera- 
tive to transform the velocity (v) of the workpiece 
(25) Into a velocity at which a potential surface 
defect (1 14, 122) Is moving In relation to said imag- 
ing device (24) through the use of said model data, 
and to predict a location of the potential surface 
defect (114, 122) from said first set of image data 
(32) to said second set of image data (34) through 
the use of said velocity (v). 

3. TTie surface inspection system of Claim 1 or 2, 
characterized in that the workpiece (25, 120) is 
modeled using a plurality of facets, where each 



facet represents a portion of the surface of the 
workpiece (25, 120), and said model data is indica- 
tive of the spatial relationship between each facet of 
the workpiece (25. 120) and the observation plane 
5 of said imaging device (24). 

4. The surface inspection system of Claim 3, charac- 
terized in that said tracking subsystem (30) is oper- 
ative to access model data which corresponds to 

10 the portion of the surface of the workpiece (25, 1 20) 
being inspected by the surface inspection system 
(10). 

5. The surface inspection system of any of Claims 1 to 
15 4, characterized by means for adjusting an intensity 

of said light source (22) based on the color of the 
surface of the workpiece (25). 

6. The surface inspection system of any of Claims 1 to 
20 5, characterized In that said anomaly detection sub- 
system (28) includes: 

a subtraction module (36) receiving said first 
set of image data (32) and said second set of 

25 Image data (34) and operative to generate 

Image data (38) indicative of the differences 
between said first set of Image data (32) and 
said second set of image data (34); and 
a thresholding module (40) receiving the image 

30 data (38) from said subtraction module (36) 

and operative to generate a binary representa- 
tion (42) of said Image data. 

7. The surface inspection system of any of Claims 1 to 
35 6, characterized in that said anomaly detection sub- 
system (28) further includes: 

an edge detection module (52) receiving said 
first set of image data (32) and said second set 
40 of image data (34) and operative to Identify the 

edges of the line of light (23) in each set of 
Image data (32, 34); 

a second thresholding module (58) receiving 
the first and second sets of Image data (54, 56) 

45 from said edge detection module (52) and 

operative to generate a binary representation 
for each set of image data; 
a dilation module (60) receiving said first and 
second sets of image data from said second 

50 thresholding module (58) and operative to 

thicken the edges of the line of light (23) in 
each set of image data; and 
a mask formation module (110) receiving said 
first and second sets of Image data (62, 64) 

55 from said dilation module (60), said mask for- 

mation module (110) operative to combine said 
first set of image data (62) with said second set 
of image data (64) into a resultant set of image 
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data using an OR operation (66) and to invert 
the resultant image data (68) using a NOT 
operation (70), thereby forming a mask (72) for 
filtering out a portion of the noise in the image 
data. 5 

8. The surface inspection system of any of Claims 1 to 
7, characterized in that said anomaly detection sub- 
system (28) further includes: 

10 

an anomaly identification module receiving 
image data (42) from said thresholding module 
(40) and a mask (72) from said mask formation 
module and operative to apply said mask (72) 
to said Image data (42) using an AND opera- is 
tion (74), thereby filtering out a portion of the 
noise in the image data. 

9. A method of detecting defects in a surface of a 
workpiece (25, 120) using a surface inspection sys- 20 
tern (10), comprising the steps of: 

emitting at least one line of light (23) onto the 
surface of the workpiece (25, 120); 
passing the surfeice of the workpiece (25» 1 20) 25 
through the line of light (23); 
capturing a first set of Image data (32) repre- 
sentative of a portion of the surface of the work- 
piece (25, 120) using an imaging device (24), 
said imaging device (24) having a plane of 30 
observation and being positionabte at a van- 
tage point such that the line of light (23) is 
within its plane of observation; 
providing model data indicative of the spatial 
relationship between said portion of the surface 35 
of the workpiece (25, 120) and the observation 
plane of said imaging device (24); 
capturing a second set of image data (32) rep- 
resentative of said portion of the surface of the 
workpiece (25, 120) using said imaging device 40 
(24); 

identifying at least one potential surface defect 
(114, 122) in said second set of image data 

(34); 

capturing a third set of image data representa- 45 
five of said portion of the surface of the work- 
piece (25, 120) using said imaging device (24); 
and 

tracking the potential surface defect (114, 122) 
from said second set of image data (32) to said so 
third set of image data using said model data, 
thereby assessing if the potential surface 
defect (114, 122) constitutes a defect in the 
surface of the workpiece (25, 120). 

55 

10. A method of detecting defects in a surface of a 
workpiece (25, 120), comprising the steps of: 



emitting at least one line of light (23) onto the 
surface of the workpiece (25, 1 20); 
passing the surface of the workpiece (25. 120) 
through the line of light (23); 
capturing a first set of image data (32) repre- 
sentative of a portion of the surface of the work- 
piece (25, 120) using an imaging device (24), 
said Imaging device (24) having a plane of 
observation and being positionabte at a van- 
tage point such that the line of light (23) Is 
within its plane of observation; 
identifying at least one potential surface defect 
(114, 122) In said first set of image data (32); 
determining a velocity (v) for the workpiece (25, 
120); 

transforming the velocity (v) of the workpiece 
(25, 120) into a velocity (v') at which potential 
surface defects (114, 122) are moving in rela- 
tion to said imaging device (24); 
capturing a second set of image data (34) rep- 
resentative of said portion of the surface of the 
workpiece (25, 120) using said imaging device 
(24); and 

tracking the potential surface defect (1 14, 122) 
from said first set of image data (32) to said 
second set of Image data (34) using the veloc- 
ity (v') at which potential surface defects (114, 
122) are moving in relation to said Imaging 
device (24), thereby assessing if the potential 
surface defect (1 1 4, 1 22) constitutes a defect in 
the surface of the workpiece (25, 120). 

1 1 . The method of Claim 9 or 1 0, further comprising the 
step of adjusting an intensity of a light source emit- 
ting said at least one line of light (23) based on the 
color of the surface of the workpiece (25, 120). 

12. The method of any of Claims 9 to 11, wherein the 
step of identifying at least one potential surface 
defect (114, 122) further comprises: 

subtracting (36) said first set of Image data (32) 
from said second set of image data (34) to form 
an intermediary set of image data (38), the 
intermediary set of image data (38) indicative 
of potential surface defects (114, 122) in at 
least said second set of Image data (32). 

13. The method of Claim 12, which further comprises 
generating a binary representation (42) of the Inter- 
mediary set of image data (38). 

14. The method of Claim 12 or 13, which further com- 
prises the step of filtering out at least a portion of 
noise in said first and second set of image data, 
where the noise is caused by a variation in the vis- 
cosity of the paint on the surface of the workpiece 
(25, 120). 
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15. The method of Claim 14, wherein the step of filter- 
ing out at least a portion of noise further comprises: 

providing a copy of said first set (32) and said 
second set (34) of image data; 5 
identifying (52) the edges of the line of light (23) 
in each copy of image data; 
combining the copy of said first set of image 
data with the copy of said second set of image 
data into a resultant set of image data by using 
an OR operation (66); 

inverting said resultant set of image data using 
a NOT operation (70), thereby forming a mask 
(72) for filtering out a portion of the noise; and 
applying said mask (72) to said intermediary 
set (42) of image data using an AND operation 
(74), thereby filtering out a portion of the noise 
in the Image data. 

16. The method of any of Claims 9 to 15, further com- 
prising: 

modeling the workpiece (25, 120) using a plu- 
rality of facets, where each facet represents a 
portion of the surface of the workpiece (25, 
120); 

providing model data indicative of the spatial 
relationship between each facet of the work- 
piece (25, 120) and the observation plane of 
said Imaging device (24); and 
tracking the potential surface defect (114, 122) 
using model data which corresponds to the 
portion of the surface of the workpiece (25, 
120) being inspected by the surface inspection 
system (10). 

17. The method of any of Claims 9 to 16, wherein the 
step of tracking the potential surface defect further 
comprises: 

determining a first location of the potential sur- 
face (1 14, 122) defect in a reference frame for 
said Imaging device (24), where the first loca- 
tion is derived from said second set (34) of 
image data; 

determining a velocity (v) for the workpiece (25, 
120); 

transforming the velocity (v) of the workpiece 
(25, 120) into a velocity (v') at which potential 
surface defects (114, 122) are moving in rela- 
tion to the said imaging device (24) through the 
use of said model data; 

predicting a location of the potential surface 
defect (1 1 4, 1 22) in said third set of image data 
by using the velocity (v') for potential surface 
defects (114.122); 

determining a second location of the potential 
surface (114, 122) defect in the reference 



frame of said imaging device (24), where the 
second location is derived from said third set of 
image data; and 

comparing the predicted location of the poten- 
tial surface defect (114, 122) with said second 
location of the potential surface defect (1 1 4, 
122); thereby tracking the potential surface 
defect (114, 122). 



15 



20 



25 



30 



35 



40 



45 



50 



9 



EP 1 092 973 A1 




EP 1 092 973 A1 




18 


0J5 






o 


m 




MS 





114 



F mre ■ 7 



11 



EP 1 092 973 A1 




EP 1 092 973 A1 



52 



, ^. 



EDGE 
DEIECim 

(mm 



BH5E 
BEmOH 
OPBtAWM 



1 i ^ V 1 



imHoum 
(mm 



imimm 
(mm 



t ^ — i 



mm 
(mm 



ttiAm 
(mm 




Fip ure - 3B 



13 



EP 1 092 973 A1 




14 



EP 1 092 973 A1 



mm 



DMsm 

OPERATION 



coHMBnim 

OFOATm 



PAIR 

EummN 

OPBUnOH 



100 



CBmOD 
DETBOmmH 
OPBiATm 



0 

F me'4A 



15 



EP 1 092 973 A1 



0 



m 


cms 
'Aim 






luvm 

OPBMTXW 






HASm 
OPBlAm 



108 



110 



FjgmjJB 



16 



EP 1 092 973 A1 




F me ' 5 



EP 1 092 973 A1 




18 



EP 1 092 973 A1 




19 



EP 1 092 973 A1 




EP 1 092 973 A1 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application NumlMr 

EP 00 12 1807 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with mdieaSon. where apprapriate. 
of relevant passages 



Relevant 
todaim 



CtASSFICATION OF THE 
AmjCATION (lnLCI.7) 



Y 

A 



Y 
A 



Y 
A 



US 5 734 742 A (SUZUKI YUTAKA ET AL) 
31 March 1998 (1998-03-31) 

* abstract * 

* coliimn 3, line 25 - line 42 * 

* column 4, lltie 19 - line 24 * 

* column 5, line 41 - column 6, line 8 * 

* figures 1,4,6A-6B,7 * 

US 5 963 328 A (USUI YOSHITAKA ET AL) 
5 October 1999 (1999-10-05) 

* column 16, line 43 - line 50 * 

« column 16, line 60 - colunn 17, line 7 • 

* colunn 17, line 55 - column 18, line 18 
* 

* column 5, line 47 - line 59 * 



WO 97 39339 A (AUTOSPECT INC) 
23 October 1997 (1997-10-23) 

* page 12, line 19 - page 15, line 

* figure 1 * 



3 * 



WO 99 04248 A (ALDERS KLAUS ;LEHE MARTINA 
(DE); WAN GANG (DE); AUDI NSU AUTO UNIO) 
28 January 1999 (1999-01-28) 

* page 6, paragraph 3 * 

* page 17, line 2 - line 5 * 

* figure 1 * 



1,6,10, 
12,13 

7,9. 
15-17 



1.6,10. 

12,13 

2.4.5,11 



G01N21/88 

G06T7/00 

GOlBll/30 



1,6.10, 

12,13 

2.4,5,11 



1.9 



Tl« prasem search report has baen drawn up Ibr an daims 



TECHNICAL FIELDS 
SEARCHED <li1t.Ck7) 



GOIN 
G06T 
GOIB 



Plaoaaf nttich 

THE HAGUE 



DbIb oI offnpWm of ttw March 

31 January 2001 



Verdoodt, E 



CATEGORY OF CfTED DOCUMENTS 

X : particulafly retovanllf taken atone 

Y : particulafly relevant if combined ¥vlth another 

document dttie same category 
A : tadtnotogical bad«9round 
O : noo-wrtten dtadosure 
P: 



T: theoiy or prtndpto undertylng the hveniion 
E : eartier patent documenl. but puWishod on, or 

aner the fling dale 
0 : document ctod In the appication 
L : document dtod fbr other teasons 



& : member o( the same patent fan%, corresponding 
documenl 



21 



EP1 092 973A1 



ANNEX TO THE EUROPEAN SEARCH REPORT 
ON EUROPEAN PATENT APPLICATION NO. 



EP 00 12 1807 



This annex lists the patent family meinl>er$relatlno to the patent documents dtsd In the above-mentioned European search report 
The membere are as contained m me European Patent Office EOP file on 

The European Patent Office Is In no way liable tor these particulars which are merely given tor the purpose of infortnatton. 

31-01-2001 



Patent document 
died in search report 


Publication 
date 


Patent family 
meinber(s) 


publication 
date 


US 5734742 


A 


31-03-1998 


JP 
KR 


8145906 A 
180082 6 


07-06-1996 
15-05-1999 


US 5963328 


A 


05-10-1999 


JP 
JP 


11072439 A 
11108641 A 


16-03-1999 
23-04-1999 


MO 9739339 


A 


23-10-1997 


CA 
EP 


2251323 A 
0898702 A 


23-10-1997 
03-03-1999 


WO 9904248 


A 


28-01-1999 


DE 
EP 


19730885 A 
0995108 A 


21-01-1999 
26-04-2000 



2 



111 Ftor more dataOs about this annex: see Offlctal Journal of the European Patent Office. No. 12/82 



22 



